Set add_special_tokens to false by default in Encode#1442
Set add_special_tokens to false by default in Encode#1442RyanUnderhill merged 4 commits intomainfrom
add_special_tokens to false by default in Encode#1442Conversation
|
Ok, meeting conclusion is that we don't need this API currently as our internal default values will do what users want. This way we avoid exposing an option that nobody knows what value to set to. |
sayanshaw24
changed the title
add_special_tokens to false by default in Encode
Co-authored-by: Ryan Hill <38674843+RyanUnderhill@users.noreply.github.com>
RyanUnderhill
left a comment
RyanUnderhill
left a comment
There was a problem hiding this comment.
Not sure if you want to say in the PR comments why we default to false, just to have some history for it if we forget why we did this in the future.
|
Sets `add_special_tokens` from `OrtxTokenizeWithOptions` added in microsoft/onnxruntime-extensions#940 to false to solve chat template issue in GenAI with extra BOS tokens. See huggingface/transformers#37686 for more context. --------- Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com> Co-authored-by: Ryan Hill <38674843+RyanUnderhill@users.noreply.github.com>
Update version to 0.8.0-rc2 and cherry pick these 3 changes: #1435 update ESRP settings #1434 make WebGPU name consistent #1432 Missed an all lowercase "webgpu" string #1440 Apply provider name backwards compatibility at runtime #1452 Update Extensions Commit to Support Chat Template Override for Unsupported Models #1439 Sign macos binaries #1442 Set `add_special_tokens` --------- Co-authored-by: Guenther Schmuelling <guschmue@microsoft.com> Co-authored-by: Sayan Shaw <52221015+sayanshaw24@users.noreply.github.com> Co-authored-by: Baiju Meswani <bmeswani@microsoft.com> Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com> Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
3 participants
Sets
add_special_tokensfromOrtxTokenizeWithOptionsadded in microsoft/onnxruntime-extensions#940 to false to solve chat template issue in GenAI with extra BOS tokens.See huggingface/transformers#37686 for more context.